Skip to content

Fix loader correctness and global side-effects from PR #103 review - #121

Merged
garlontas merged 1 commit into
bugfix/#95/loading-big-data-files-not-safefrom
copilot/modify-all-loaders
Apr 12, 2026
Merged

Fix loader correctness and global side-effects from PR #103 review#121
garlontas merged 1 commit into
bugfix/#95/loading-big-data-files-not-safefrom
copilot/modify-all-loaders

Conversation

Copilot AI commented Apr 12, 2026

Copy link
Copy Markdown
Contributor

Addresses all unresolved review comments on the lazy-loader refactor: a correctness bug in JSON parsing, a global shared-state bug in the XML loader, and test hygiene issues.

JSON loader

  • yield from on non-array top-level object was wrong. json.loads(..., object_hook=...) returns a namedtuple for a top-level {} — iterating over it with yield from unpacks its field values, not the object itself. Now checks isinstance(result, list) and yields the single object directly when the root is not an array.
  • Whitespace-only files now handled correctly. src == '' replaced with not src.strip() in the file reader path.
# Before — yields 'value1', 'value2' instead of the namedtuple
yield from jsonlib.loads('{"key1": "value1", "key2": "value2"}', object_hook=...)

# After — yields the namedtuple as a single item
result = jsonlib.loads(src, object_hook=__dict_to_namedtuple)
if isinstance(result, list):
    yield from result
else:
    yield result

XML loader

  • Removed module-level config object. Mutating config.cast_types / config.retrieve_children at call time is not safe under concurrent use. Both options are now passed as explicit arguments through the full call chain (_lazy_parse_xml_file, _lazy_parse_xml_string, _parse_xml_string_lazy, and all __parse_* helpers).

Tests

  • test_xml_loader.py: Renamed mock_csv_filemock_xml_file; updated docstring; added setUp to initialize self.file_content and avoid a latent AttributeError when content is not passed.
  • test_yaml_loader.py: Added test_yaml_loader_with_malformed_yaml asserting yaml.YAMLError is raised for invalid input.

Summary by Sourcery

Fix JSON and XML loader behavior while tightening related tests.

Bug Fixes:

  • Correct JSON loader to yield a single top-level object when the JSON root is not an array.
  • Treat whitespace-only JSON input as empty when loading from files or strings.
  • Eliminate shared mutable configuration in the XML loader by passing retrieve_children and cast_types options explicitly through the parsing pipeline.

Enhancements:

  • Refine XML loader internals to take parsing options as function parameters instead of relying on a module-level utility object.

Tests:

  • Rename and clean up XML loader test helpers, adding a setUp initializer to prevent latent attribute errors.
  • Add a YAML loader test to assert malformed YAML raises yaml.YAMLError.

Agent-Logs-Url: https://github.com/pickwicksoft/pystreamapi/sessions/df7a64c6-e671-45e1-adf6-ae09cef8ec2a

Co-authored-by: garlontas <70283087+garlontas@users.noreply.github.com>
@sourcery-ai

sourcery-ai Bot commented Apr 12, 2026

Copy link
Copy Markdown

Reviewer's Guide

Refines JSON and XML loader behavior to fix iteration correctness and remove shared mutable state, and updates XML/YAML loader tests to match the new semantics and improve robustness.

Sequence diagram for XML loader call chain with explicit flags

sequenceDiagram
    actor Caller
    participant xml
    participant _lazy_parse_xml_file
    participant _lazy_parse_xml_string
    participant _parse_xml_string_lazy
    participant __parse_xml
    participant __parse_empty_element
    participant __parse_single_element
    participant __parse_multiple_elements
    participant LoaderUtils

    Caller->>xml: xml(src, read_from_src=False, retrieve_children, cast_types, encoding)
    alt read_from_src is False
        xml->>LoaderUtils: validate_path(src)
        xml->>_lazy_parse_xml_file: _lazy_parse_xml_file(path, encoding, retrieve_children, cast_types)
        _lazy_parse_xml_file->>_lazy_parse_xml_file: open file and read xml_string
        _lazy_parse_xml_file->>_parse_xml_string_lazy: _parse_xml_string_lazy(xml_string, retrieve_children, cast_types)
    else read_from_src is True
        xml->>_lazy_parse_xml_string: _lazy_parse_xml_string(src, retrieve_children, cast_types)
        _lazy_parse_xml_string->>_parse_xml_string_lazy: _parse_xml_string_lazy(xml_string, retrieve_children, cast_types)
    end

    _parse_xml_string_lazy->>_parse_xml_string_lazy: root = ElementTree.fromstring(xml_string)
    _parse_xml_string_lazy->>__parse_xml: __parse_xml(root, cast_types)

    alt element has no children
        __parse_xml->>__parse_empty_element: __parse_empty_element(element, cast_types)
        alt cast_types is True
            __parse_empty_element->>LoaderUtils: try_cast(element.text)
            LoaderUtils-->>__parse_empty_element: cast value
        else cast_types is False
            __parse_empty_element-->>__parse_xml: element.text
        end
        __parse_xml-->>_parse_xml_string_lazy: parsed_value
    else element has one child
        __parse_xml->>__parse_single_element: __parse_single_element(element, cast_types)
        __parse_single_element->>__parse_xml: __parse_xml(sub_element, cast_types)
        __parse_xml-->>__parse_single_element: sub_item
        __parse_single_element-->>_parse_xml_string_lazy: namedtuple_single
    else element has multiple children
        __parse_xml->>__parse_multiple_elements: __parse_multiple_elements(element, cast_types)
        loop over each child e
            __parse_multiple_elements->>__parse_xml: __parse_xml(e, cast_types)
            __parse_xml-->>__parse_multiple_elements: parsed_child
        end
        __parse_multiple_elements-->>_parse_xml_string_lazy: namedtuple_multiple
    end

    alt retrieve_children is True
        _parse_xml_string_lazy-->>Caller: yield from __flatten(parsed)
    else retrieve_children is False
        _parse_xml_string_lazy-->>Caller: yield parsed
    end
Loading

Flow diagram for JSON loader top-level result handling

flowchart TD
    A[Start JSON load] --> B[Read JSON source - file or string]
    B --> C{json_string.strip is empty?}
    C -->|Yes| D[Return without yielding any items]
    C -->|No| E[Call jsonlib.loads with object_hook __dict_to_namedtuple]
    E --> F{Is result a list?}
    F -->|Yes| G[Iterate over result and yield each item]
    F -->|No| H[Yield result as a single item]
    G --> I[End]
    H --> I[End]
Loading

File-Level Changes

Change Details Files
Make JSON loader correctly handle non-array root objects and whitespace-only input for both file and string sources.
  • Replace empty-string check with a whitespace-trimming emptiness check before parsing JSON content.
  • Capture the result of jsonlib.loads with object_hook applied instead of iterating directly over it.
  • Branch on whether the parsed JSON root is a list; yield from it when it is, otherwise yield the single parsed object.
  • Apply the same parsing and yielding logic in the string-based JSON loader path.
pystreamapi/loaders/__json/__json_loader.py
Remove global XML loader configuration and pass parsing options explicitly through the XML parsing call chain.
  • Delete the __XmlLoaderUtil helper class and its module-level config instance.
  • Change the public xml() API to forward retrieve_children and cast_types flags into the lazy parsing helpers instead of mutating global state.
  • Extend _lazy_parse_xml_file, _lazy_parse_xml_string, and _parse_xml_string_lazy to accept retrieve_children and cast_types parameters.
  • Thread cast_types into __parse_xml and its helpers so type casting is controlled entirely by arguments rather than global config.
  • Use retrieve_children flag directly when deciding whether to flatten parsed XML children.
pystreamapi/loaders/__xml/__xml_loader.py
Align XML loader tests with XML-specific helpers and ensure proper setup of shared test state.
  • Introduce setUp to initialize self.file_content used by XML loader tests.
  • Rename the mock context manager from mock_csv_file to mock_xml_file and update its docstring to reference XML instead of CSV.
  • Update all XML loader tests to use the new mock_xml_file context manager.
tests/_loaders/test_xml_loader.py
Add regression test for malformed YAML handling in the YAML loader.
  • Introduce a malformed YAML string in tests.
  • Assert that yaml_lib.YAMLError is raised when loading malformed YAML via yaml(..., read_from_src=True).
tests/_loaders/test_yaml_loader.py

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@garlontas
garlontas marked this pull request as ready for review April 12, 2026 17:39
@garlontas
garlontas merged commit 150cd58 into bugfix/#95/loading-big-data-files-not-safe Apr 12, 2026
4 of 5 checks passed
@garlontas
garlontas deleted the copilot/modify-all-loaders branch April 12, 2026 17:39

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 2 issues, and left some high level feedback:

  • In TestXmlLoader.mock_xml_file, if the implementation still uses content = content or self.file_content, then test_xml_loader_with_empty_file will never actually pass an empty string ('' is falsy) and will instead use self.file_content; consider switching to a sentinel check (e.g., if content is None: content = self.file_content) so the empty-content test behaves as intended.
  • The JSON loader’s file and string paths now duplicate the same strip/jsonlib.loads/isinstance(list) logic; consider extracting this into a shared helper to keep the behavior consistent and ease future changes.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- In `TestXmlLoader.mock_xml_file`, if the implementation still uses `content = content or self.file_content`, then `test_xml_loader_with_empty_file` will never actually pass an empty string ('' is falsy) and will instead use `self.file_content`; consider switching to a sentinel check (e.g., `if content is None: content = self.file_content`) so the empty-content test behaves as intended.
- The JSON loader’s file and string paths now duplicate the same `strip`/`jsonlib.loads`/`isinstance(list)` logic; consider extracting this into a shared helper to keep the behavior consistent and ease future changes.

## Individual Comments

### Comment 1
<location path="tests/_loaders/test_xml_loader.py" line_range="103-106" />
<code_context>
             self.assertRaises(StopIteration, next, data)

     def test_xml_loader_is_iterable(self):
-        with self.mock_csv_file(file_content):
+        with self.mock_xml_file(file_content):
             data = xml(file_path)
             self.assertEqual(len(list(iter(data))), 3)

     def test_xml_loader_with_empty_file(self):
</code_context>
<issue_to_address>
**suggestion (testing):** Consider adding tests for XML loader when using `read_from_src=True` with different flag combinations

With the refactor removing the global `config` and threading flags through the call chain, the `read_from_src=True` path (`_lazy_parse_xml_string`) should be covered similarly to the file-path loader.

Could you add tests that:
- Call `xml(xml_string, read_from_src=True)` with `retrieve_children` set to both `True` and `False`, checking the parsed data shape matches the existing file-based tests.
- Call `xml(xml_string, read_from_src=True, cast_types=False)` and assert numeric/boolean-like values remain strings, mirroring `test_xml_loader_no_casting`.

You can reuse the existing `file_content` XML string so both entry points (`file_path` and `read_from_src`) stay aligned without much extra test code.

Suggested implementation:

```python
    def test_xml_loader_no_casting(self):
        with self.mock_xml_file(file_content):
            data = xml(file_path, cast_types=False)

            first = next(data)
            self.assertRaises(StopIteration, next, data)

    def test_xml_loader_from_src_retrieve_children_true(self):
        # read_from_src=True should behave the same as the file-path loader
        with self.mock_xml_file(file_content):
            file_data = list(xml(file_path, retrieve_children=True))

        src_data = list(xml(file_content, read_from_src=True, retrieve_children=True))
        self.assertEqual(src_data, file_data)

    def test_xml_loader_from_src_retrieve_children_false(self):
        # read_from_src=True with retrieve_children=False should mirror file-path behaviour
        with self.mock_xml_file(file_content):
            file_data = list(xml(file_path, retrieve_children=False))

        src_data = list(xml(file_content, read_from_src=True, retrieve_children=False))
        self.assertEqual(src_data, file_data)

    def test_xml_loader_from_src_no_casting(self):
        # read_from_src=True with cast_types=False should mirror test_xml_loader_no_casting
        with self.mock_xml_file(file_content):
            file_data = list(xml(file_path, cast_types=False))

        src_data = list(xml(file_content, read_from_src=True, cast_types=False))
        self.assertEqual(src_data, file_data)

    def test_xml_loader_is_iterable(self):
        with self.mock_xml_file(file_content):
            data = xml(file_path)
            self.assertEqual(len(list(iter(data))), 3)

    def test_xml_loader_with_empty_file(self):
        with self.mock_xml_file(''):
            data = xml(file_path)
            self.assertRaises(ParseError, next, data)

```

These changes assume:
1. The `xml` loader already accepts `read_from_src` and `retrieve_children` keyword arguments, matching the refactor you mentioned.
2. `file_content` is the XML string used in other tests in this module, and `file_path`/`mock_xml_file` are available helpers as shown.

If there are existing dedicated tests for `retrieve_children=True/False` with the file-path loader earlier in this file, these new tests now assert that the `read_from_src=True` path produces identical output to the file-based path for the same flags, without needing to duplicate structure-specific assertions.
</issue_to_address>

### Comment 2
<location path="tests/_loaders/test_yaml_loader.py" line_range="67-72" />
<code_context>
             data = yaml(file_path)
             self.assertIsInstance(data, GeneratorType)

+    def test_yaml_loader_with_malformed_yaml(self):
+        malformed_yaml = "key: : invalid"
+        with self.assertRaises(yaml_lib.YAMLError):
+            list(yaml(malformed_yaml, read_from_src=True))
+
     def _check_extracted_data(self, data):
</code_context>
<issue_to_address>
**suggestion (testing):** Extend malformed YAML coverage to the file-based loader path

You’ve covered the `read_from_src=True` path. To keep behavior consistent with the other loader tests, please add a file-based variant that:

- Writes the same malformed YAML to a temp file (or uses the existing file-mocking helper).
- Calls `yaml(file_path)` without `read_from_src=True`.
- Asserts `yaml_lib.YAMLError` is raised when the generator is consumed.

This will verify that both string and file inputs fail the same way for invalid YAML.

```suggestion
    def test_yaml_loader_with_malformed_yaml(self):
        malformed_yaml = "key: : invalid"
        with self.assertRaises(yaml_lib.YAMLError):
            list(yaml(malformed_yaml, read_from_src=True))

    def test_yaml_loader_with_malformed_yaml_file_path(self):
        malformed_yaml = "key: : invalid"
        with patch(PATH_EXISTS, return_value=True), \
             patch(PATH_ISFILE, return_value=True), \
             patch(OPEN, mock_open(read_data=malformed_yaml)):
            with self.assertRaises(yaml_lib.YAMLError):
                list(yaml("malformed.yaml"))

    def _check_extracted_data(self, data):
```
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Comment on lines 103 to 106
def test_xml_loader_is_iterable(self):
with self.mock_csv_file(file_content):
with self.mock_xml_file(file_content):
data = xml(file_path)
self.assertEqual(len(list(iter(data))), 3)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (testing): Consider adding tests for XML loader when using read_from_src=True with different flag combinations

With the refactor removing the global config and threading flags through the call chain, the read_from_src=True path (_lazy_parse_xml_string) should be covered similarly to the file-path loader.

Could you add tests that:

  • Call xml(xml_string, read_from_src=True) with retrieve_children set to both True and False, checking the parsed data shape matches the existing file-based tests.
  • Call xml(xml_string, read_from_src=True, cast_types=False) and assert numeric/boolean-like values remain strings, mirroring test_xml_loader_no_casting.

You can reuse the existing file_content XML string so both entry points (file_path and read_from_src) stay aligned without much extra test code.

Suggested implementation:

    def test_xml_loader_no_casting(self):
        with self.mock_xml_file(file_content):
            data = xml(file_path, cast_types=False)

            first = next(data)
            self.assertRaises(StopIteration, next, data)

    def test_xml_loader_from_src_retrieve_children_true(self):
        # read_from_src=True should behave the same as the file-path loader
        with self.mock_xml_file(file_content):
            file_data = list(xml(file_path, retrieve_children=True))

        src_data = list(xml(file_content, read_from_src=True, retrieve_children=True))
        self.assertEqual(src_data, file_data)

    def test_xml_loader_from_src_retrieve_children_false(self):
        # read_from_src=True with retrieve_children=False should mirror file-path behaviour
        with self.mock_xml_file(file_content):
            file_data = list(xml(file_path, retrieve_children=False))

        src_data = list(xml(file_content, read_from_src=True, retrieve_children=False))
        self.assertEqual(src_data, file_data)

    def test_xml_loader_from_src_no_casting(self):
        # read_from_src=True with cast_types=False should mirror test_xml_loader_no_casting
        with self.mock_xml_file(file_content):
            file_data = list(xml(file_path, cast_types=False))

        src_data = list(xml(file_content, read_from_src=True, cast_types=False))
        self.assertEqual(src_data, file_data)

    def test_xml_loader_is_iterable(self):
        with self.mock_xml_file(file_content):
            data = xml(file_path)
            self.assertEqual(len(list(iter(data))), 3)

    def test_xml_loader_with_empty_file(self):
        with self.mock_xml_file(''):
            data = xml(file_path)
            self.assertRaises(ParseError, next, data)

These changes assume:

  1. The xml loader already accepts read_from_src and retrieve_children keyword arguments, matching the refactor you mentioned.
  2. file_content is the XML string used in other tests in this module, and file_path/mock_xml_file are available helpers as shown.

If there are existing dedicated tests for retrieve_children=True/False with the file-path loader earlier in this file, these new tests now assert that the read_from_src=True path produces identical output to the file-based path for the same flags, without needing to duplicate structure-specific assertions.

Comment on lines +67 to 72
def test_yaml_loader_with_malformed_yaml(self):
malformed_yaml = "key: : invalid"
with self.assertRaises(yaml_lib.YAMLError):
list(yaml(malformed_yaml, read_from_src=True))

def _check_extracted_data(self, data):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (testing): Extend malformed YAML coverage to the file-based loader path

You’ve covered the read_from_src=True path. To keep behavior consistent with the other loader tests, please add a file-based variant that:

  • Writes the same malformed YAML to a temp file (or uses the existing file-mocking helper).
  • Calls yaml(file_path) without read_from_src=True.
  • Asserts yaml_lib.YAMLError is raised when the generator is consumed.

This will verify that both string and file inputs fail the same way for invalid YAML.

Suggested change
def test_yaml_loader_with_malformed_yaml(self):
malformed_yaml = "key: : invalid"
with self.assertRaises(yaml_lib.YAMLError):
list(yaml(malformed_yaml, read_from_src=True))
def _check_extracted_data(self, data):
def test_yaml_loader_with_malformed_yaml(self):
malformed_yaml = "key: : invalid"
with self.assertRaises(yaml_lib.YAMLError):
list(yaml(malformed_yaml, read_from_src=True))
def test_yaml_loader_with_malformed_yaml_file_path(self):
malformed_yaml = "key: : invalid"
with patch(PATH_EXISTS, return_value=True), \
patch(PATH_ISFILE, return_value=True), \
patch(OPEN, mock_open(read_data=malformed_yaml)):
with self.assertRaises(yaml_lib.YAMLError):
list(yaml("malformed.yaml"))
def _check_extracted_data(self, data):

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants